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ABSTRACT 


During the 2005 North Atlantic hurricane season, an 
objective EroOpiéa cyclone vortex LOETIOWELCation and 
tracking technique was applied to analyzed and forecast 
fields of three global operational numerical models - the 
Nationad Centers for Environmental Prediction Global 
Forecast System (GFS), the Navy Operational Global 
Atmospheric Prediction System (NOGAPS), and the United 
Kingdom Meteorological Office model (UKMET). For the 
purpose of evaluating each model’s performance with respect 
to forecasting tropical cyclone formation, 14 relevant 
parameters are cataloged for every tropical vortex. 

In this study, nine of the fourteen parameters are 
subjected to a linear discriminant analysis applied to all 
forecast vortices that exceed vorticity and warm core 
thresholds. The goal is to determine the combination of 
parameters for each model, at each 12-h forecast period to 
120h, that best discriminates between a vortex that 1s 
correctly forecast to intensify into a tropical cyclone 
(developer) and a vortex that is forecast to intensify into 
a tropical cyclone, but does not (false alarm). The 
performance of the resulting discriminant functions are 
then assessed using the Heidke Skill Score and Receiver 
Operating Characteristic curves. Overall, the methodology 
applied to forecasts from the UKMET model shows the most 
Skill with regard to identifying correct forecasts of 


Eropical Syclone: TOPrMathon. 
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1 Ae INTRODUCTION 


Improvements a ai operational global models have 
resulted in increased accuracy of 72-h tropical cyclone 
track forecasts. In an effort to extend preparation lead- 
Gime; “the National Hurricane ‘Center (NHC) and ~Ehe Jornet 
Typhoon Warning Center (JTWC) began issuing track forecasts 
through <IZO0m ih ZOOS. DLNCe <a “ErOopical -eyclone could 
easily form and intensify to a powerful storm within this 
120-h window, an increased need exists for accurate 
prediction of tropical cyclone formation from operational 
global models. While it 1S important to establish the 
accuracy “Ob forecasts: “of Lropiucal. cyclone formation, 12 
Global models, 26 as. -also. amportant; to: @dentity and 
understand factors that distinguish forecasts of vortices 
that are correctly forecast to intensify into tropical 
cyclones (developers) from forecasts of vortices that are 
hOrecast LO: Gmbensify into: “bropical -cyclones;, -but -do. Nor 
(false alarms). 

A. DEFINITION OF FORMATION 

The criteria. for. defining “tropical cyclone formation 
are not universally accepted, although all definitions 
require that a tropical cyclone be convective and non- 
frontal, originate over tropical waters, have a cyclonic 
surface circulation, and have a warm core with winds that 
are strongest at the surface and decrease with height. 
Flsberry (2003) defines formation to have occurred when 


NW 


there exists av MOMSEPON tal, Cyclone jCieeulacnonm. 21: -Ehe 
tropics that is closed (ground-relative westerly winds on 
the equatorward side for a westward moving storm) with 


maximum 10-minute averaged surface sustained winds of at 


beast. 25 ke. (1925 m Go") “hat. ie aeccomanied by dees 
(throughout most of the troposphere) convection and a 
radius of maximum winds such that the Rossby number is at 
least one.” A Rossby number of at least one would require 
a radius of maximum wind less than 500 km for a maximum 
wind of 12.5: am 6 @&. 70° Tacieude:. Tes. <errverion. 1s 
included to distinguish between a tropical cyclone, which 
has maximum winds near the center, and a monsoon depression 


that has maximum winds at a larger radius. 


Davis and Bosart (2003) use the requirement that the 
vortex be capable of self-amplification through air-sea 
interaction. This occurs around the time that the cyclone 
reaches wind speeds of 34 kt (17 mS) @t- which point the 
character of the air-sea interaction changes such that 
there are enhanced fluxes of heat and moisture between the 
ocean and atmosphere. Ths 28 “the 2ypical “Ss LOmm: 
threshold. Most operational centers, including the NHC and 
JTWC begin issuing advisories when the vortex meets the 
\depression” threshold: of 25 ke (12: m Ss) «Bor ‘the purpose 
of this study, formation will be considered to have 
occurred at the time of the first advisory issued by the 
NHC or JTWC, which as also the time of its first entry in 
the “Destrirack® -<database: 


B. FAVORABLE ENVIRONMENTAL CONDITIONS FOR TROPICAL 
CYCLONE FORMATION 


Gray (1975) cited six physical parameters that create 
a favorable, though not sufficient, environment for the 
formation of tropical cyclones. They are: (1) large values 
of low-level relative vorticity; (ti) va. WOCarLoOn Of, “ab 
least 500 km from the equator (to gain some contribution 
£Yom. CorLelis:. Eoree) + (111) weak vertical shear of the 


horizontal winds; (iv) sea-surface temperatures exceeding 
2 


26°C through a deep oceanic layer; (v7) conditional 
instability through a deep atmospheric layer; and (vi) a 
moist lower- and mid-troposphere. The first three describe 
dynamical aspects of the environment while the last three 


relate to its thermodynamic state. 


DeMaria et al. (2001) concluded that tropical cyclone 
formation in the Atlantic 1S constrained by vertical 
instability and midlevel moisture during the early part of 
the hurricane season, and by vertical wind shear in the 
latter part of the season. Molinari et al. (2000) 
demonstrated that because environmental conditions in the 
eastern North Pacific are almost always favorable for 
Formation; 2b. 28 “the amplituae: of approaching tropical 
waves that is the most important factor determining whether 
or not tropical cyclones will form. 

C. TROPICAL WAVES 

The most common pre-existing disturbance, or formation 
seedling, in both the Atlantic and eastern North Pacific is 
the tropical, or easterly, wave (Figure 1). A westward- 
moving wave train develops in the trade winds’ and 
convergence on the eastern side of the trough leads to 
COnveclLon:. The tropical wave that affects these regions 
1s the African Easterly Wave, which develops from 
baroclinic-barotropic instability in the mid-level easterly 
jet over Africa. In the Atlantic, 40% of hurricanes and 
65% of major hurricanes originate as tropical waves in the 
area between North Africa and the Caribbean (DeMaria et al. 
ZOD) 4 Avila and Pasch (1995) state that almost all 
tropical cyclones in the eastern North Pacific can be 


traced to an African Easterly Wave. 





Figure l A schematic of a wave in the easterly trade 
winds (From: NASA Earth Observatory Library 
http://earthobservatory.nasa.gov/Library/Hurri 
canes; June 2006). 


D. FORMATION MECHANISMS 

While the environmental conditions required LO 
formation are understood, and the importance of ae pre- 
existing disturbance to organize convection and concentrate 
potential vorticity (Davis and Bosart 2001) 1S recognized, 
the processes that cause this disturbance to develop into a 
tropical cyclone are not well understood. Enhanced global 
models and higher resolution satellite imagery have 
improved the forecast process Significantly, but 
insufficient “in-situ” observations have been available 
over the tropical oceans to verify hypotheses and confirm a 


Single mechanism of formation. 


In general, if the favorable environmental conditions 
are met, an area of convection experiences positive 
feedback in that warm air rises, condenses and releases 


latent heat. That heat further increases the buoyancy of 
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the air. In response to warming in the rising air, the 
surface pressure falls and leads to a rapid inflow of air, 
which, in turn, leads to more thunderstorms. COLrVoda:s 
force will eventually cause the low-level inflow to develop 
ai "CV GLOMILG:. iCatieCULa tasons Warming of the rising air causes 
alr pressure to rise at the top of the storm. A ane 
cyclonic outflow develops that provides an exhaust for the 
storm and allows the above sequence of events to continue. 
E. USE OF GLOBAL MODELS TO FORECAST FORMATION 

As described by Gray CLOTS. both dynamical and 
thermodynamic conditions may create a favorable environment 
for tLOpieal cyclone EOrMat LOM. The favorable 
thermodynamic COnOTELens SxLSet over broad geographic 
regions, change slowly, and are fairly well resolved by 
global models. However, the dynamical conditions vary on a 
much smaller temporal and Spatial scales. Not 
Surprisingly, 2t as. therefore much «more ditfiacult for 


global models to resolve changes in these variables. 


Due to these limitations, some studies (e.g., Beven 
1999) concluded that the global operational forecast models 
hac: “no Signi tveant. Skill “im Torecasting “tropical cyclone 
FOEMeE LOM. Improved horizontal and vertical resolution in 
the models as well as remotely sensed observations has 
helped global models better resolve the physical processes 
on smaller and smaller scales. However, Hennon and Hobgood 
(2003) suggest that global models could provide skillful 
formation forecasts even if complex mesoscale processes are 
neglected. They found several characteristics associated 


with the large-scale environment over the North Atlantic 


that may provide significant measures to discriminate 
between cloud clusters that develop into tropical cyclones 
and those that do not develop. 
F. A GENESIS PARAMETER 

Gray (LO7S) used. the product © his: cynamic. Criteria 
(low-level relative vorticity, distance from the equator, 
and vertical shear of the horizontal winds) and the product 
of the thermodynamic criteria (sea-surface temperature, 
COnditaonal instability and -bower= <and: mid=Cropospheric 
moisture) to define a Seasonal Genesis Parameter. DeMaria 
et al. (2001) developed a similar tropical cyclone genesis 
parameter for the Atlantic that was the product of vertical 
shear, vertical TPS CaO da Ary and midlevel moisture 


variables. 


Using eight predictors from large-scale analyses, 
Hennon and Hobgood (2003) developed a probabilistic 
prediction system for tropical cyclone formation. They 
manually tracked cloud clusters in satellite imagery and 
then submitted the eight variables associated with each 
cluster to 2: disecramanen’e analysis: The resulting linear 
combination of the variables best separates cloud clusters 
that will develop into tropical depressions from those that 
will not. 

G. PLAN FOR THE THESIS 

In this thesis, a discriminant analysis similar to 
that used by Hennon and Hobgood (2003) 1s applied to 
forecasts OF EOD rCad: vortices produced by several 
operational global models. Vortices are identified and 
tracked based on their analyzed and forecast vorticity as 
opposed to the Hennon and Hobgood approach of identifying 


candidate eloud clusters zion a satellite imagery. A 


probability-based model 1s used to define the likelihood 
that a tropical vortex that is forecast by the operational 
model to become a tropical cyclone will actually develop 
into a tropical cyclone. Whereas Hennon and Hobgood (2003) 
used the discriminant analysis approach for the broader 
problem of distinguishing between developing and non- 
developing cloud clusters, this study is limited to cases 
in which a vortex is forecast to develop into a tropical 
cyclone, and the objective is to provide the probability 


that the model forecast is true or false. 


The Vortex Tracking (VORTRACK) system of Harr (2006) 
is the key to this type of data analysis. Therefore, the 
first part of Chapter II (Methodology) will describe the 
VORTRACK system. The data analysis approach will then be 
summarized. For each operational global model and forecast 
interval considered, the approach ImClUeGee- produeiig 
SCabLter plots, «probability “GrStrabuLron Tunecrions (PDFs); <a 
linear discriminant analysis (which included a jackknife 
routine fOr application KO independent Gata ) 5 and 
performance assessment to include the Heidke Skill Score 


(HSS) and Receiver Operating Characteristics (ROC) curves. 


In Chapter III, the results will be presented, which 
will aiunclude the parameters used in each discriminant 
function along with relevant PDFs, as well as the HSS and 
ROC curve Tor each cdrscriminent Tuncreizon.« Chapter IV will 
summarize the conclusions, highlight some of the challenges 
posed by aga: type of data analysis, and make 


recommendations for further study. 
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II. METHODOLOGY 


To discriminate between operational model forecasts of 
developing and non-developing vortices, it is necessary to 
extract parameters that are relevant to tropical cyclone 
formation relative to each tropical vortex. These 
parameters must be identified in forecasts of varying 
lengths for comparison with the parameters in the verifying 
analyses. Furthermore, it iS important to capture these 
characteristics among several operational models. The 
database defined by the VORTRACK system, which 1s 
summarized in Figure 2, 1s used to identify forecast model 
parameters that have the most predictive value with regards 


tO 2ermaliron. 


The principal objective of this research is to create 
such a tool that can discriminate between a developer and a 
false alarm in the global model forecasts. This tool will 
be based on a probabilistic assessment derived from a 
linear discriminant analysis. 
A. VORTRACK 

The VORTRACK application was developed at the Naval 
Postgraduate School (NPS) so that output from numerical 
models could more easily be included in the process of 
Forecasting: the 2nrtensiticacvon; Of <a wropical dow 1nko. <a 
tropical storm. A prototype was run for the 2005 hurricane 
season at both NPS and the NHC. There are two primary 
components of VORTRACK. The first is the data processing 
segment which is the Tropical Cyclone Vortex Tracking 
Program (revirP) . THES program. .ourputs “che: tracks: of -adt 
eligible tropical vortices as well as the corresponding 


values of 14 environmental parameters. The resulting ASCII 
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text file is stored in one location if the vortex can be 
matched in space and time with a vortex in the verifying 
analysis field, and in another location if the vortex 
appears only the forecast fields. The second component of 
VORTACK 1S a web-interface output segment (Harr 2006). 
Vortex Locations and environmental parameter 
characterizations from the TCVTP-generated ASCII text files 


were used for this thesis. 


Forecast Model Analyses and Forecast Fields 


Tropical Cyclone 
Vortex Tracking 
Program(ICVTP) 


Forecast Data Base Analvzed Data Base 
Circulations Current tropical 
forecast to circulations 
occur 
Current circulation 

Catalog of summaries! verifications 
False Alarms 

Final Circulation Catalog 

sum maries/verifications 





Figure 2 Schematic of the TCVTP component of VORTRACK 
(From: Harr 2006). 

Ds Tropical Cyclone Vortex Tracking Program (TCVTP) 
The objective tropical vortex identification and 
tracking technique, TCVTP, was utilized to examine analyzed 
and forecast fields of three operational global numerical 
models during the 2005 hurricane season in the North 
Atlantic. The models that had been catalogued in the 2005 
database include the Navy Operational Global Atmospheric 
Prediction system (NOGAPS), the National Centers EOT 
Environmental Prediction’s Global Forecast System (GFS), 
and the United Kingdom Meteorological Office (UKMET) global 


model. 
Lo) 


For each tropical vortex, which is defined by the 
850 hPa relative vorticity above a threshold value of 
1.0 x 10° 6) 14. environmental warameters im model. analyses 
and forecasts that are relevant to tropical cyclone 
formation (Table 1) are catalogued in the TCVTP. In this 
study, only nine of the 14 parameters were used. Those 
parameters shaded on Table 1 were not used in this study. 
For example, the 850-500 hPa average relative vorticity was 
chosen over the 850 hPa relative vorticity. The 700- 
400 hPa, TOO 300 hPa warm COre, and precipitation 
parameters were not used because these values were not 
available for the UKMET model and therefore they could not 


be used for model inter-comparison. 


Table 1 Analyzed and forecast quantities used _ to 
identify physical characteristics associated 
with each romance ak vortex. Warm core 
measurements are defined aS a temperature 
difference between the vortex and the 
environment (From: Harr 2006). 

SO nba Redra bikye a Vor eile by Sea-level pressure 


Minimum (mb) 


Shallow Vertical Wind Shear Deep Vertical Wind Shear 
(S590=50.0 nea) (850-200 hPa) 


850-200 hPa Geopotential 7O00-500 hPa Warm Core 


Height Thickness 


Vertical motion at 500 hPa 700-400 hPa Warm Core 
700-500 hPa Vapor Pressure TOU —3 CO Vinegar Wearcme € One 


850-500 hPa Average Relative | Sea-level Pressure Difference 


VOrEDCITy between the Vortex and the 


PAVironment 
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B. SOME DEFINITIONS 

The characterization of a vortex as developer or non- 
developer was based on the 850-500 HPA average relative 
vorticity and 700-500 hPa warm core values aS compared to 
threshold values for these parameters. The threshold value 
(Pr) for each parameter is defined as the lowest tercile of 
that parameter averaged for all developers at the time of 
LOrMaT VOr< Time of formation (T 9) is defined as the time 
that the tropical cyclone was first entered in the best- 
track database. A developer is any tropical vortex that 
was eventually entered in the best-track database. For the 
purpose of this study, this included all forecast hits 
(vortex was forecast to exceed the threshold values and the 
corresponding vortex in the verifying analysis was a 
developer) and misses (vortex was not forecast to exceed 


the threshold values but was in fact a developer). 


A non-developer is defined as a vortex tracked in the 
TCVTP database that never entered the NHC best-track 
database, which may include three sub-categories. A false 
alarm 1S a tropical vortex that was forecast to exceed both 
threshold values, but 1s actually a non-developer because 
it did not verify above those threshold values. The 
majority of vortices in the TCVTP database was not forecast 
to exceed both threshold values and did not verify above 
the threshold, values; these correct nulls were not 
considered in this study. The third category consists of 
VOCELCES. im: “EChe YECWEPRP “that ‘were forecast ©O be; and 
according to model analyses verified above, both threshold 
values but were never entered in the NHC best-track 
database. In these cases, there were other parameters not 


considered in this study (e.g., low-level wind shear) that 


TZ 


must have prevented these vortices from becoming 
developers. Even though they meet the definition of non- 
developers, since the forecast verified in the correct 
category (1.e., upper-right quadrant of the scatterplots in 
Appendix A), they are not regarded as false alarms and were 
also not considered in this study. 
C. PRELIMINARY ANALYSIS 

Since by definition any tropical cyclone must have 
positive lower-tropospheric relative vorticity and a warm 
core, all tropical vortices in the TCVTP database for each 
model at each forecast interval were plotted on a graph 
with 850-500 hPa average relative vorticity on the x-axis 
and 700-500 hPa warm core on the y-axis. Figure 3 1s an 
example of this scatterplot for the GFS 12-h model data. 
Two clusters were expected: developers with a centroid in 
the upper-right Guadhake, and non-developers with a 
centroid in the lower-left quadrant. As evident on 
Figure 3, such a separation between developers and non- 


developers in the model forecasts was not observed. 


This and scatterplots at longer forecast intervals 
demonstrated that the GFS model does not have skill in 
distinguishing between developers and non-developers when 
considering the two most basic prereguisites of tropical 
cyclones (positive relative vorticity and a warm core). 
Similar results were achieved with the NOGAPS and UKMET 
models at all forecast periods. As a result, the aim of 
this study was shifted to discrimination between developers 
and false alarms. Since a false alarm is a non-developing 
vortex with forecast values of 850-500 hPa average relative 
vorticity and 700-500 hPa warm core above threshold values, 


these false alarms also are in the upper-right quadrant on 
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Figure 3. That is, only developers entered in the best- 
track data and non-developers in the upper-right quadrant 
of the vorticity / warm core scatterplots are considered in 


the following sections. 


(SF 12-h Forecast Avg. 50-500 hPa VYorticity and 7U0-500 hPa Vvarm Core 





700-500 hPa Warm Core (°C) 


HoU-5U0 hPa Relative Vorticity qo sy 


Figure 3 A scatterplot of 850-500 hPa average relative 
vorticity and 700-500 hPa warm core for all 
vortices in the TCVTP database for the GFS 
model at the 12-h forecast time. The dark 
solid lines represent the calculated threshold 
values (for this model and forecast period) 
for these two parameters. 


For this GFS 12-h example, the vortices of interest 
are identified in Figure 4. On this figure, black squares 
represent developers. Solid circles represent false 
alarms. Cyan represents vortices that were false alarms in 


warm core only (te tej verified above the VOrCLieiLy 
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threshold). Magenta represents vortices that were false 
alarms in vorticity only (i.e. verified above the warm core 
threshold). Green represents vortices that were false 
alarms in both warm core and vorticity. The objective of 
the linear discriminant analysis is then to distinguish 
between the developers (squares on Figure 4) and false 


alarms (solid colored circles on Figure 4). 
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Figure 4 As in Figure 3 except with non-developers, 
developers and three kinds of false alarms 
annotated as indicated in the inset. 


The next step in the data analysis was to include 
additional parameters on these scatterplots. iim Frogure. oy 
vortices are plotted as in Figure 4, except that vortices 
with 500 hPa omega values above the threshold value for 


that parameter are indicated (with a blue versus’ black 
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Square in the case of developers and with a triangle versus 
circle in the case of false alarms). The few open red 
circles in the upper right quadrant represent vortices that 
verified above the thresholds for both warm core and 


vorticity and were therefore not considered false alarms. 


(SF 12-h Forecast Avg. g50-500 hPa Yorticity and 7U0-500 hPa Vvarm Core 





Now Dene bop 

Deve hop 

Deve bp aid = ongea 
Fancy 

FAQS) 2+ omega 
FA quory 

FAdon &< omega 
Fv ebotiy 

FApotiyS = omega 


hee 
in OP 
dias es i 
See Se ayy le 
‘a — 


a ; tale at 
Ms ies 


. ad es “oH 
ad a 


700-500 hPa Warm Core (°C) 


H5U-5U0 hPa Relative Vorticity qo sy 


PiOuUre 5 As in Figure 4, except with vortices exceeding 
the threshold for the 500 hPa vertical motion 
(omega) parameter annotated as indicated in 
the inset. 


The goal in completing these scatterplots was to 
determine which parameters had the best potential for 
discriminating between developers and false alarms and thus 
to reduce the number of parameters considered in the 
di sCriminant analysis. However, InSpeCL Lon On the 


scatterplots revealed no systematic tendencies associated 
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with various parameters that drstianguashed between 
developers and false alarms. Therefore PDFs of each 
parameter and vortex category were considered and are 
described in the next sub-section. For 2@nrOrmation 
purposes, the scatterplots including each of the seven 
parameters in the GFS model at 12h are displayed in 
Appendix A, and the scatterplots including omega in all 
three models at all ten forecast intervals are displayed in 
Appendix B. The scatterplots in Appendix B are useful for 
illustrating the number of cases considered in each test, 
which helps account for some of the problems encountered in 
the data analysis at the longer forecast intervals. 

D. PROBABILITY DISTRIBUTION FUNCTIONS (PDFS) 

Probability Density Functions (PDFs) were completed 
for the nine parameters in each model at each forecast 
interval and are contained in Appendix C. An example from 
the UKMET 12-h forecast data is provided in Figure 6. The 
goal in completing the PDFs was to reduce the number of 
parameters to be considered in the discriminant analysis. 
It was assumed that those parameters that had the smallest 
amount of overlap in the blue and red curves would be the 
best discriminators between developers and false alarms. 
Whereas this may be true when considering an individual 
parameter IS became evident aon) preliminary linear 
discriminant analyses that the combinations of parameters 
behaved in ways that were not obvious from the individual 
plots (e.g., Figure 6). Thus, a more comprehensive 
analysis of all combinations of parameters needed to be 
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Figure 6 Probabiiary Distriburion Funcrions of 
developers in green, developers in blue, 


false alarms 


in red for 


the nine parameters 
derived from the UKMET 12-h forecast relative 


to tropical vortices in the TCVTP. 


E. LINEAR DISCRIMINANT ANALYSIS 


The goal of a 


discern group membership based on a vector 
attributes that are observed for n cases 


training sample that contains n cases is used to define the 


discriminant 
specified a priori 
only one group. 
cases 


developers and false 


lee grea eniae aime 


in the training sample. 


linear 


The 


alarms are the 
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disc rininent 


number 


In this 


analysis 
(Xx) 
(Wilks, 


OE «Groups 


Group membership must be specified for all 


study, g=2 


two GLOUpS, 


having k 
2 05)) « 


(g) 


and each case must belong to one and 


and k=9 


Since nine parameters from the CEP database are 
considered. The training sample size (n) varies by model 


and forecast interval. 


The discriminant analysis process consists of two 
segments. Discrimination 1s the process of using the 
training: date ‘Co: LFand’ a. £uncrLlon -oOr Zhe attributes that 
best discriminates between the groups (the discriminant 
FUNCELON): « Classification is the process of assigning new 
cases to one of the groups using the discriminant function. 
Alternately, classification may involve estimating the 
probabilities of group membership. iy “this “Stucy,. bork 
methods of classification were also applied to independent 
data using a ‘jackknife’ technique to test the skill of the 
discriminant function. This process will be described in a 


later section. 


A discriminant function is needed for each model and 
forecast interval (30 in total). For a given model and 
FOrecast ianterval, “the data Matrix (fh x k) Can be -<divided 
into the two groups that result in two matrices with 
dimensions (n, x k) and (no x k) where n = ni i No. The 
number of developers (n;) and false alarms (nz) in the model 
data varies by model and forecast period. Hence, the first 
matrix has dimensions of (n, xX 9) and the second matrix has 
dimensions (no xX YQ). TheGoal. £S* tO: Pic... near Funclwvon 
6f the kk parameters. or ‘the Timear Combination ax that. will 
best classify the group membership of future cases. Tas 


linear combination is defined as the discriminant function. 


Assuming the two populations corresponding to the 
groups have the same covariance structure, Wilks (2005) 
Summarizes the approach taken by statistician R.A. Fisher 


as finding “the vector a as that direction in the _ k- 
ie, 


dimensional space of the data that maximizes the separation 
of the two means, in standard deviation units, when the 
data are projected onto a.” This procedure requires that a 


maximizes 


als.h ' (1) 
a. Dada 


where x, and x2, are the mean vectors for each group and Spool 
1s the estimated common covariance matrix of the two groups 
which is a weighted average of the two sample covariance 
matrices 


n, -l n, —l 
—_]§ —+—_|8, |. 
RE we a a) 


The advantage of finding the direction a is that the 
data vector, x, can be transformed into a scalar variable, 
5i=a’x, which is known as Fisher’s linear discriminant 
function as defined above. The groups of multivariate data 
are hence transformed to # £groups of univariate data 
distributed along the @ axis. The vector a, which lies in 


the direction of maximum separation is given by 
Seen ee (3) 
and Fisher’s linear discriminant function becomes 
6, =a x=(x, ~x,)"|S uf 'x. (4) 


This function maximizes the scaled distance (D) between the 
two sample means in the training data where it is defined 


lower 
a’ (xX, —X,) =(X, -x,)'[S,.a]° @, -X,) =D’. (2) 


Zn 


Future cases will be classified as belonging to either 
Group 1 or Group 2 according to the value of 6;. Because 6; 
VS :a -dOt product, 2: 2S a Scalar, or One-dimensional 
projection of the vector x onto the direction of maximum 
separation, a. The simplest method of classification is to 
assign an individual case in x, to Group 1 if the 
OFOTJECEIGON ax is closer ‘to the projection of the Grove: 1 
mean and assign it to Group 2 if it is closer to the 


projection of the Group 2 mean. 
The midpoint between the means of the two groups along 
the a@ axis 1S given by: 


ho lee ae = l l : 
m = 5X +a'X,) = 5a +X,)= mae =X) [S57] 6 Saeco (6) 


Therefore a case in X is classified according to the 


following rule: 
Assign to-Group lata x2m 


ASsion EOGroup 2 it a x <M 


Alternately, a probabilistic classification can be 
done. Using this method, the probability that a case in x 
belongs to each of the two groups is determined according 


to Bayes’ Theorem: 


pf, (&) 
Pr\Group l|x > = -———+-+\+>—_ ‘| 
ps) pf, (x) + p.f, («) ma 
Pr{Group 2\x = Paty) (8) 


pf, (x) + p,f, (x) ) 


In Equations (7) and (8), the p; and pz values are the prior 
probabilities of group membership, which is simply the 
relative frequency with which group 1 and group 2, 


respectively, are represented in the training data. Thus fj 
pail 


and f2 are the PDFs for groups 1 and 2, respectively. If it 
can be assumed that the PDFs are multivariate normal 


distributions, Equations (7) and (8) become: 


ey 
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al l as Ii = _ 
a 2expl-—(x-X,) [82] a) 
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iin ehas Sead yy. Ene: -probabmility Ehreshold sat. twhavch:.. <a 
case was classified as belonging to one group or the other 
was incrementally adjusted between O and 1. This process 
1s explained Chapter II, Section G which describes the ROC 
curves. 
F. JACKKNIFE ROUTINE 

A Jackknife routine eliminates one case at a time and 
recalculates the relevant statistics (Wilks 2005). eng 
example, the first case can be removed from the data matrix 
which results in a matrix with dimensions ([n-l1] x k). The 
linear discriminant analysis is performed on this matrix 
and the resulting discriminant function can then be used to 
classify the excluded case. The process is repeated by re- 
inserting the first case and removing the second case, and 
SO: “On This routine provides a method for testing the 
function with independent data in the absence of a new 
dataset. 
G. HEIDKE SKILL SCORE (HSS) 

Forecast verifications are often placed in contingency 


tables similar to Table 2. 
OW, 


Table 2 A contingency table in which forecast status is 
compared with observed status. Hits are 
represented by a, false alarms by b, misses by 
Cy. ane (COrrece Wwe soy <0: 


Observed 
a ee 
Pe | | 
Yes 





No 
According the Wilks (2006), the reference accuracy 
measure in the HSS is the proportion of random forecasts 
(statistically independent of the observations) that would 
be correct. Perfect forecasts receive a score of 1 and 
forecasts equivalent to random chance receive a score of O. 


If the forecasts are worse than random forecasts, they 


receive a negative score. The probability of a correct 


NW LA 


yes” forecast by chance is equal to the probability of a 


\ AA 


yes forecast times the probabilicy Or a “yes” 


ObServalion, Or 





(a+b) (atc) (a+b)(atc) 


nN Nn n° 


(ih) 


and the probability of a correct “no” forecast is 


(b +d) (c +d) Z (b+c)(c +d) 


nN Nn n° 





(22) 


These definitions are equivalent to the assumption 
that the forecasts and observations are independent. The 
HSS is then equal to the ratio of the number correct to the 
number incorrect by chance, divided by one minus the number 


correct by chance, which simplifies to: 


2(ad — bc) 


HSS == 22 T7Y]__71_____ 
(a+c)(c+d)+(a+b)(b+d) 


(13) 
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The Hos 2S. £irsc calbculatrec for discriminant. Luncrerons 
containing rs el possible combinations of the nine 
parameters. The discriminant function that shows the most 
skill is then submitted to the jackknife procedure. 

H. RECEIVER OPERATING CHARACTERISTICS (ROC) CURVES 

The use of Receiver Operating Characteristic (ROC) 
curves originated in the signal processing community. In 
this sense, the threshold for which a signal is determined 
to be present is incrementally adjusted between the point 
at which a signal is always detected and the point at which 
a signal is never detected. The probability of detection, 
or hit rate (equivalent to d/(ctd) in Table 2), and the 
false-alarm rate (equivalent to b/(atb) in Table 2) are 
calculated for each of these thresholds. These points are 
plotted on a graph with the false-alarm rate (0 to 1) along 
the x-axis and the hit rate (0 to 1) along the y-axis. The 


resulting points. form the ROC Curve. 


A ROC curve that followed a diagonal from (0,0) to 
(1,1) would indicate that the ‘receiver’ produced as many 
false alarms as hits. The area under the curve in this 
case would be 0.5. The ideal ROC curve would follow along 
the left and top axes which would indicate that the 
receiver had a perfect hit rate and produced no false 
alarms. The area under this curve would be 1. In reality, 
most receivers fall somewhere between these two extremes. 
The closer a ROC curve is to the upper left corner, the 
closer the receiver 1s to ideal (all hits and no false 
alarms). Theretore,. the erea under the -curve (AUC) -1.s one 


measure of performance. 


For purpose of this study, the threshold for which a 


candidate vortex was classified as being a developer was 
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varied. By default, this number is 0.5. Leg: Jase: On. “Ene 
jyackknifed discriminant FUNC E LOM, a vortex has 
classification scores of 0.51 for Group 1 (developers) and 
OG49: .£6r° -Group 2 (false alarms) Le 26° BocoOmeticadl ly 
classified as a developer. By allowing the threshold to 
vary, some degree of fuzziness iS introduced into the 
analysis and an additional measure of skill, the Area Under 
tne “Curve (AUC), becomes available. Furthermore, “the 
classification threshold value that produces the point on 
the ROC curve that is farthest from the diagonal is chosen 
as the optimal threshold for the applicable model and 


forecast period. 
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III. RESULTS 


A. PRESENTATION OF DISCRIMINANT ANALYSIS STATISTICS 

Table 3 31s divided into ten blocks for the ten 
forecast intervals with a row for each of the three models. 
The first and second columns list the model and forecast 
period (tau), respectively. The third column lists the 
predictors that were used in the applicable discriminant 
function (Table 4 lists the parameters that corresponds to 
the numbers in this column). TRE: POuUrth: and: TiEeth eo lumns 
Contain -two -Hos: values. The first score is for the 
discriminant function using dependent data and the second 
score 1s Lor the discriminant Function using the 
independent data. Both skill scores were calculated for a 
ChasstErCartonm Bareshold “or. U5. The sixth column gives 
the Area Under the Curve (AUC) for the ROC curves displayed 


in the subsequent figures. 


The next section includes a summary of the results for 
each forecast interval. One of the key results is the 
parameters that are common to the discriminant functions 
applied to each model. The relevant HSS and ROC curves are 
also summarized. In the summary of 12-h forecasts, a 
detailed example of the methodology as applied to the UKMET 
12-h model data will be provided. The last section of this 


chapter will describe two classification case studies. 


Z4 


Table 3 Results of the Linear Discriminant Analysis. 


Model Tau Predictors Used Maximum HSS Jackknife HSS AUC 
GFS 12h Te + Ae 5 0.5924 0.2917 0.5362 





Ly ee pe Op ye emer love: C209 72 0.59.00 
0.0000 0 093 033000 








Table 4 List of parameters corresponding to numbers in 
ColuM 3-0 Tabe’-<3. 


650=500 
hPa hPa 


Average Vapor 


VOCEICLEY Pressure 





Wise: 


B. DISCUSSION OF RESULTS 
A 12-h Forecasts 

a. Application of Methodology to UKMET Forecast 

A total of 19 developers and 31 false alarms were 
identified in the scatterplot for the UKMET 12-h model data 
(see Appendix B). The pertinent matrices for developers 
(19 x 9) and false alarms (31 x 9) were submitted to a 
series of linear discriminant analyses that considered all 
possible combinations of the nine parameters. Using a 
contingency table (as in Table 2), the HSS was calculated 
for each of the discriminant functions and the function 
that resulted in the highest HSS was selected. In the UKMET 
12-h case, a discriminant function using seven of the nine 
parameters (850-500 hPa Average Vorticity, 700-500 hPa Warm 
Core, 850-500 hPa Shear, 500 hPa Omega, 700-500 hPa Vapor 
Pressure, SLP Difference, and SLP Minimum) exhibited the 


most. -okadcil:. 


Table 5 indicates that only five cases were 
misclassified by the discriminant function applied to the 
UKMET 48-h forecast. Three cases did develop despite being 
classified as false alarms and two cases did not develop 
despite being classified as developers. These results, as 
applied to the calculation of the HSS, are more simply 
displayed in a contingency table (Table 6), which can also 
be used to compute hit and false-alarm rates for use in a 


ROC diagram. 


Z9 


The next step was to perform a jackknife routine 
on the UKMET 12-h forecast data and recalculate all 
relevant statistics. As expected, the contingency table 
for the discriminant function using the independent data 
(Table 7) has less skill (HSS=0.55) compared to that uSing 
dependent data (HSS=0.79). 


Table 5 CLassit1¢Gartton results EO the relevant 
discriminant function as applied to dependent 
data from the UKMET 12-h model. Column one 


lists the actual group membership of each 
vortex; a ‘1’ represents a developer anda ‘2’ 


represents a false alarm. Column two lists the 
group to which each vortex was assigned by the 
GVSscrimainais  huncre ron. The probabilities, 


according GO “Lhe discriminant Function,  Ehat 
the vortex belongs to group ‘1’ and ‘2’ are 
listed in columns three and four, respectively. 
Shaded rows represent vortices that were 
misclassitied by <‘the> Gdiseriminant. Lunctron, 
The three light grey rows represent vortices 
that did develop, but were classified as false 
alarms. The two dark grey rows’ represent 
vortices that did not develop, but were 
classified as developers. 


Observed | Forecast Group 1 Group 2 
Prsenved [Foreseet | probebiity | probebinity 
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Observed | Forecast Group 1 Group: 2 
Preemved [Fowesset | probebiiey | probebinity 
ee = 
po 00998 | 0 9002 
Pot 097059038 
poe 00946 0 9054 
200285 0 9TS 
2088520 6A 
Po en 5338 


DO} DO TROT RO] N |] DO 
DO }DO TDN} DOT NM] NO] DO 





0.9547 0.0453 
Table 6 Classification results from Table 5 summarized 
as a forecast contingency table. In this case, 


the HSS is 0.79. 


Observed Observed 

Developer | Non-developer 
POrSCaG Ll 16 
Developer 





Forecast 
PE SE 
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Table 7 Forecast contingency table for the relevant 
discriminant function as applied to independent 
data from the UKMET 12-h model. tm sehacs: «Case; 
the: HSS) 8.0255 


Observed Observed 
Developer | Non-developer 


Goel! se | 
Developer 
Forecast 
4 ve 
EE SEE 


In both of the above scenarios the skill scores 





were calculated using a classification threshold of 0.5. 
For the purpose of calculating hit and false-alarm rates to 
generate ROC curves, contingency tables were produced for 
the independent data by adjusting the classification 
threshold from 0 to 1 by increments of 0.1 (Table 8). At a 
threshold of zero, every case is classified to be a 
developer, and at a threshold of 1.0, every case is 
classified as a false alarm. The hit and false-alarm rates 
resulting from the thresholds between these two extremes 
were plotted to form a ROC curve (Figure 7). In this UKMET 
12-h case, the point plotted for the threshold of 0.4 1s 
the farthest from the diagonal and is therefore regarded as 


the optimal threshold. 
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Table 8 Forecast contingency tables for the relevant 
discriminant function as applied to independent 
data from the UKMET 12-h model. Fach two-row 
block gives the results produced using the 
threshold listed in column 1. The last column 
gives the resulting hit and false-alarm rates 
that were used to plot a ROC curve for the 
diSCrLMLnNane une GALOn.. 


acuakta Observed | Observed 
resho Developer | Non—Deve looper 


Forecast 
Developer 
Forecast 
False alarm 
FOLeECast 
Developer 
Forecast 
False alarm 
Forecast 
Developer 
Forecast 
False alarm 
FOCECaASt 
Developer 
FOreCcast 
False alarm 
FOreCcase 


Developer 
Forecast 
False alarm 


FOrecast 
Developer 
Forecast 
False alarm 
POrecast 
Developer 
Forecast 
False alarm 
Forecast 
Developer 
EPOreCast 
False alarm 
Forecast 
Developer 
FOrecast 
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Hit Rate: 
19/19 
False-alarm 
Rate: 31/31 
Hit Rate: 
19/19 
False-alarm 
Rate: 19/31 
Hit Rate: 
18/19 
False-alarm 
Rate: 11/31 
Hit Rate: 
16/19 
False-alarm 
Rate: 8/31 
Hit Rate: 
16/19 
False-alarm 
Rate: 8/31 
Hit. Rate: 
leap alles) 
False-alarm 
Rate: 7/31 
Hit Rate: 
is gal) 
False-alarm 
Rate: 4/31 
Hit Rate: 
dP de Q 
False-alarm 
Rate: 3/31 
Hit Rate: 
Oy wiles, 
False-alarm 





Observed Observed 
ls 
lice eee Non-Developer SS 
False alarm Races 27 31. 
Forecast Hit Rate: 


Developer 9/19 
Forecast False-alarm 


False alarm . Rate: 2/31 
Forecast 0 Hat. Rave: 
Developer 0/19 
Forecast False-alarm 
False alarm Rate: 0/31 





b. Results for 12-h Forecasts 


At Zh, . “Ene CGrseriminant vLUneCeLoOns: sapplied. .6-..dadd 


models (Table 3) include VOCE MEI ey and warm core 
parameters. The functions applied to GFS and NOGAPS also 
include the thickness parameter. Functions applied to the 


NOGAPS and UKMET models include the 850-500 hPa shear and 
both SLP parameters. The highest skill score is attained 
by the application of the discriminant function to the 
UKMET model. This score is 0.79 using dependent data 
(compared to Oe a, and 0.60 EOre GFS and NOGAPS, 
respectively) and 0.55 using independent data (compared to 
0.29 and 0.25 for GFS and NOGAPS, respectively). The ROC 
curve for the UKMET function (Figure 7) also has the 
largest. AUC (Ow/3S> compared “ro: 0.53. dnd. 0.55 for “GES -and 
NOGAPS, respectively) with an Opt mma: classir Caton 
threshold of 0.4. Therefore, the methodology applied to 
forecasts from the UKMET model, as described in the 
previous section, results . the most accurate 
discrimination between developing vortices and false 


alarms. 
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ROC Curve: 12-h Forecasts 


Hit Rate 


——— Gs 





—e— UKM 
—#— NoGaPS 
o o4 o2 o3 o4 os 0.6 7 8 og 1 
False Alarm Rate 
Figure 7 ROC curves for 12-h forecasts based on an 
independent dataset. The curve for. the 


function applied to GFS is displayed in red, 
UKMET in green, and NOGAPS in blue. 
2. 24-h Forecasts 
For the 24-h forecast interval, the discriminant 
functions applied to each of the three sets model forecasts 
(Table 3) again include the warm core parameter, but only 
the GFS function includes vorticity. All functions also 
include 850-500 hPa shear and the SLP difference. The 
functions applied to the GFS and NOGAPS models include a 
thickness parameter while the GFS and UKMET functions 
include the omega and vapor pressure parameters. The 
functions for the NOGAPS and UKMET models include SLIP 


minimum. 
The discriminant function applied to the UKMET model 


data again has the highest skill in identifying developing 
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vortices using both dependent data (HSS equal to 0.65 
compared to 0.60 for GFS and 0.53 for NOGAPS) and 
independent data (HSS equal to 0.42 compared to Q.33 for 
GFS and 0.26 for NOGAPS. The ROC curve (Figure 8) for the 
UKMET model again has the largest area (AUC equal to 0.64 
Compared to. 0.58 for GFS and i060 for -NOGAPS) with an 
optimal classification threshold of 0.4. Overall, the 
discriminant analysis applied to the UKMET forecast 1s 
marginally more successful aia identifying developing 
vortices than that applied to NOGAPS or GES. 

33 36-h Forecasts 

All three 36-h forecasts resulted in discriminant 
functions that included the warm core and omega parameters 
(Table 3). Im ddLeton, <unerLons: -applwed, to ihe: GES: and 
NOGAPS models include vorticity, 850-200 hPa shear, and SLP 
minimum. The functions applied to the NOGAPS and UKMET 


models also utilize the thickness parameter. 
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ROC Curve: 24-h Forecasts 


Hit Rate 





O o.4 o.2 o.3 o4 O.5 0.6 OF o.8 o.9 1 
False Alarm Rate 


Figure 8 ROC curves as in Figure 7, except for 24-h 
forecasts. 


As for the 12-h and 24-h forecasts, the function 
applied to the UKMET model data has the most skill in 
discriminating between developing vortices and false 
alarms. While the skill scores for the functions using 
dependent data were not significantly different among the 
three models (0.51, 0.58, 0.63 for GFS, NOGAPS and UKMET, 
respectively), the skill scores uSing independent data had 
a much larger range (0.19, 0.28, 0.43, respectively). The 
AUC for the UKMET ROC curve (Figure 9) 1s 0.60 which is 
only slightly more than that for NOGAPS (0.59) and GES 
(0.57). The optimal threshold for the UKMET function is 
O.5. Therefore, the discriminant function applied to UKMET 
model forecasts again exhibits marginally increased skill 
in identifying developing vortices. 


Md 


ROC Curve: 36-h Forecasts 


Hit Rate 





O o.4 o2 o.3 o4 O.5 0.6 OF o.8 o.9 1 
False Alarm Rate 


Figure 9 ROC curves as in Figure 7, except for 36-h 
forecasts. 
4. 48-h Forecasts 


For the 48-h forecasts, the discriminant functions 
applied to all models once again include the warm core 
parameter (Table 3). The functions for the GFS and NOGAPS 
models include the SLP difference. The functions for the 
NOGAPS and UKMET models include vapor pressure and SLP 
minimum. The functions for the GFS and UKMET models include 


VOreVCiLiy. 


The function applied to the UKMET forecast again 
attained the highest skill score. The HSS was 0.80 when 
applied to dependent data and 0.56 when applied to 
independent data. By comparison, these scores were 0.48 
and 0.35 for the GFS function and 0.65 and .034 for the 
NOGAPS Tunction, The ROC curve for the UKMET function 
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(Figure 10) also has the largest area. The AUC was 0.75 as 
compared to 0.60 for the GFS function and Q.55 For the 
NOGAPS function. The optimal classification threshold for 
the UKMET function was 0.5. Overall at 48 hours, the 
methodology applied to forecasts from the UKMET model 
results in the most accurate discrimination between 


developing vortices and false alarms. 


ROC Curve: 48-h Forecasts 


Lf 


Hit Rate 
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False Alarm Rate 


Figure 10 ROC curves, aS in Figure 7, except for 48-h 


forecasts. 


ome 60-h Forecasts 


All functions for the 60-h forecasts include vorticity 
and vapor pressure (Table 3). The functions applied to the 
GFS and NOGAPS forecasts include the SLP difference while 
the functions applied to the NOGAPS and UKMET forecasts 


include the thickness parameter and the SLP minimum. 
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The discriminant function applied to the UKMET model 
has significantly more skill in identifying developing 
vortices than the other functions. Using dependent data, 
the HSS was 0.93 as compared to only 0.13 and 0.39 for the 
functions applied to the GFS and NOGAPS models. Using 
independent data, the UKMET score was 0.45 as compared to 
O.16 and 0.07 for the functions applied to the GFS and 
NOGAPS models. While the ROC curves (Figure 11) are not as 
widely spread as the skill scores, the curve for the UKMET 
function (with an optimal classification threshold of 0.6) 
still has an AUC of 0.73 compared to only 0.55 and QO.51 for 
GFS and NOGAPS, respectively. Overall, the discriminant 
analysis applied to the UKMET model is significantly more 


successful in identifying developing vortices. 


ROC Curve: 60-h Forecasts 


Hit Rate 





O o.4 o.2 o.3 o4 O5 0.6 OF o.8 o.9 1 
False Alarm Rate 


Figure 11 ROC curves, as in Figure 7, except for 60-h 
forecasts. 
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6. 72-h Forecasts 

Using 72=h model. forecasts, all -discriminant Lunctions 
include the warm core and 850-200 hPa shear (Table 3). The 
functions applied to the GFS and NOGAPS models include the 
SLP minimum. The functions applied to the GFS and UKMET 
models include vorticity and 850-500 hPa shear. The 
functions applied to the GFS and NOGAPS models include 


vapor pressure. 


The function applied to the NOGAPS forecast has more 
Skill “Ghan -the’.<GEs and: URMET. ‘fumertionns.. The HSS was 0.6/7 
using dependent data (compared to 0.42 for GFS and 0.61 for 
UKMET) and 0.40 using independent data (compared to 0.16 
fOr Gro: vend Ugel. tor . UKM). While the skill scores for 
the functions using both the dependent and independent data 
Spanned a wide range, the ROC curves were closely grouped. 
The ROC curve (Figure 12) for the discriminant function 
applied to NOGAPS had the largest AUC (0.60) but this is 
only slightly more than the curves for the GFS (0.56) or 
UKMET (0.58) FURCELTONS:. The optimal ClassrEr cation 
Ghreshold. aor Jehe: NOGAPS. -Funcraon. ae: 0.6. Overall, the 
FunNCE1OnN: Tor the: NOGAPS: Torecast. has slightly more skill in 
identifying developing vortices at 72h. 

7. 84-h Forecasts 

Por -che -o4-h rorecasis,. Lhe CGrscriminent PuUnCELOnSs Lor 
all models include the warm core parameter and vapor 
pressure parameters (Table 3). The functions applied to 
the GFS and NOGAPS models include the 850-200 hPa shear. 
The functions applied to the NOGAPS and UKMET models 


TCIM Ae: Ore mere ys 
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Hit Rate 





O o.4 o.2 o.3 o4 O5 0.6 ed o.8 o.9 1 
False Alarm Rate 


Figure 12 ROC curves as in Figure 7, except for 72-h 
LOreCcascts. 


The discriminant function applied to the UKMET model 
forecasts has a e$significantly higher HSS using both 
dependent / independent data (0.81 / 0.48, respectively, 
Compared tO 0.51 7 Qe27 tor GFS @nd 0275 ~ O2822 tor 
NOGAPS) . The ROC curves (Figure 13) are closely grouped 
For the three functions, bout Che UKMET fEunction still has 
the largest AUC at 0.66 compared to 0.64 for the functions 
applied to both GFS and NOGAPS. Overall, the function 
applied to UKMET model data exhibits the most skill at 84h. 
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ROC Curve: 84-h Forecasts 





mmm mm mmm mm mm 


om mmm me 


Hit Rate 


a 


a ee Po 


mmm mmm mm mmm mmm mmm mm mm 


: 3 } : : : : —e ors 
‘ae Etats oe bree beers ene mea 
: : | | : : 7 —*— Nosars 


O o.4 o.2 o.3 o4 O5 0.6 ed o.8 o.9 1 
False Alarm Rate 


Figure 13 ROC curves as in Figure 7, except for 84-h 
LOreCcasts. 

8. 96-h Forecasts 
By 96h, the number of developing cases identified in 
each model has decreased appreciably. Only one developer 
was identified in the UKMET 96-h forecast which results in 
an unrealistic discriminant Lume Eom for UKMET that 
includes only one parameter. Although the function has a 
perfect skill score using the dependent data, when that one 
case 1s removed, the function has a skill score and AUC of 
zero uSing ‘independent’ data. The corresponding ROC 
curve is therefore not included in Figure 14. The NOGAPS 
and GFS forecasts still have enough developers to make the 


discriminant analysis well-posed, however, as the forecast 
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interval increases, the resulting statistics are presumed 
to be less meaningful given the decrease in the size of the 


data set. 


The only parameters included in the discriminant 
functions for both the GFS and NOGAPS models are omega and 
the SLP difference. The discriminant function applied to 
the GFS model (with scores of 0.50 and 0.38) exhibits more 
Skill in identifying developing vortices than that of 
NOGAPS (with scores of .32 and 0O.21). The ROC curve 
(Figure 14) for the GFS function also has a slightly larger 
AUC (0.62) than that of NOGAPS (0.55). In summary, the 
methodology applied to the GFS forecast is the most 


successful in identifying developing vortices at 96h. 


ROC Curve: $96-h Forecasts 
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Figure 14 ROC curves as in Figure 7, except for 96-h 
forecasts and the curve for the UKMET function 


1s omitted. 
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9. 108-h Forecasts 

The NOGAPS 108-h forecast had only one developing 
system, therefore the methodology does not provide 
realistic results. This ROC curve is not included in 
Figure 15. The discriminant function applied to the UKMET 
model, which is based on significantly fewer cases than the 
GFS function, attains a skill score of 1 using independent 
data while the latter attains a skill score of 0.60. The 
score for the UKMET function, however, decreases to 0.31 
using independent data, which is equal to that for the GFS 
function. The UKMET function has the largest AUC (Figure 
1S) et Us.6/, Whiten 2s only slagnrtly larger than the AUC Tor 
the function applied to the GFS model (.58). Overall, the 
functions applied to UKMET and GFS model data display 
roughly equivalent skill at 108h. 
ROC Curve: 108-h Forecasts 





Hit Rate 


O Oo. O.2 O.3 O.-} O.5 0.6 oF O.8 o.9 ‘| 
False Alarm Rate 
Figure 15 ROC curves as in Figure 7, except for 108-h 
forecasts and the curve me Bas the NOGAPS 
FUNCTION AS Omitted. 
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10. 120-h Forecasts 

Very few 120-h forecasts of developers exist in any of 
the models. Since there are only two developers in the 
NOGAPS forecast, the methodology applied to this model is 
not meaningful. The ROC curve for the NOGAPS function is 
therefore not included in Figure 16. The discriminant 
functions applied to the GFS and UKMET forecasts include 
the warm core, 850-500 hPa shear, and omega parameters 
(Table 3). The function applied to the UKMET model has 
higher skill scores (O.72 and 0.42) than the function 
applied to the GFS model (0.55 and 0.10) and the ROC curve 
(Figure 16) for the UKMET function has a larger AUC (0.65) 
Liam that. or Ene GFo Tunceron (0.55) « 


ROC Curve: 120-h Forecasts 
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Figure 16 ROC curves as in Figure 7, except for 120-h 
forecasts and the curve LO the NOGAPS 
function is omitted. 
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on CLASSIFICATION EXAMPLES 

In this section, two cases are examined to provide an 
example of how the methodology would be interpreted with 
respect to a single tropical vortex that is forecast to 
exceed vorticity and warm core thresholds. 

1 Correctly Classified False Alarm 

This section examines the model data, statistics, and 
classification of a case from the UKMET 48-h model. The 
vortex, which was tracked in the TCVTP from 17°N, 15°W at 
LZ00ULC 7] July DONS 7 was assigned the dle 12) 
UBM, Z00SCTORLZ Tet Ole” by foe. satabase: The 48-h UKMET 
forecast to be discussed was initiated at 1200UTC 14 July 
2009: The 48-h vorticity and warm core values for this 
vortex are identified on Figure 17 with a blue arrow. The 
sequence of forecast tracks for this vortex is shown in 
Figure 18. The 48-h track forecast is illustrated in 


Figure 19. 


The UKMET forecast and analyzed values for the six 
parameters used in the 48-h discriminant function (Tables 3 
and 4) are compared in Figures 20-25. Figure 20 
illustrates that the 48-h forecast relative vorticity was 
higher than the analyzed relative vorticity. Figure 21 
illustrates that the 48-h forecast warm core was higher 
than the analyzed warm core. Figure 22 illustrates that 
the 48-h forecast wind shear was lower than the analyzed 
wind shear. Figure 23 illustrates that the 48-h forecast 
vertical motion was higher than the analyzed vertical 
motion. Figure 24 illustrates that the 48-h forecast vapor 
pressure was higher than the analyzed vapor pressure. 
Figure 25 illustrates that the 48-h forecast sea-level 


Pressure Was: /OWwer Than the analy7ze0 Sea-level pressure, 


4] 





700-500 hPa Warm Core (°C) 


Figure 1/7 


UKM 48-h Forecast Avg. 850-500 hPa Vorticity and 700-500 hPa VVarm Core 


© Nor Deukp 
BM cewke 
@ Fawn 


950-500 hPa Relative Vorticity (10°° s"') 


Scatterplot of 850-500 hPa relative vorticity 
and 700-500 hPa warm core for all vortices in 
the TCVTP database for the UKMET model at the 


4A8-h forecast interval. The case 
UKM 2005070712 17.015 a8 haghlrghted with a 
blue arrow. The case UKM 2005091900 018 O02 


is highlighted with a green arrow. 
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Figure 18 The analyzed track (black) and all UKMET 
forecast Cracks (COLOrsS) Lor the case 
UKM 2000070712 17 O15. 
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Figure 19 As in Figure 18, except with the UKMET 
forecast track initiated at 1200 UTC 14 July 
2005 (blue) and the 48-h analyzed and forecast 
positions circled in red. 
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UKM 2005070712 _17_015: 850-500 hPa Relative Vorticity 
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Figure 20 Average relative vorticity (10 Ss) between 


850 and 500 hPa Pie case 
UKM 2005070712 17 015. The  UKMET £48-h 
forecast values from 1200 UTC 14 July 2005 are 
plotted with red circles, while analyzed 
values are plotted with blue squares. 








UKM 2005070712 _17_015: 700-500 hPa Warm Core 
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Figure 21 As in Figure 20, except for warm core (°C) at 
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UKM 2005070712_17_015: 200-850 hPa Wind Shear 
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Figure 22 As in Figure 20, except for vertical wind 
shear (ms) between 200 and 850 hPa. 








UKM 2005070712 17 015: 500 hPa Vertical Motion 
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Figure 23 As in Figure 20, except for vertical motion 
(m st * 10) at 500 hPa. 
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UKM 2005070712 17 015: 700-500 hPa Vapor Pressure 




















Vapor Pressure (mb) 















































7/7105 7/8/05 7/9/05 7/10/05 7/11/05 7/12/05 7/13/05 7/14/05 7/15/05 7/16/05 
Date 








—®— Forecast: 0000 UTC 15 July 2005 —i Analysis 

















Figure 24 As in Figure 20, except for vapor pressure 
(mb) between 700 and 500 hPa. 





UKM 2005070712_17 015: Sea-Level Pressure 
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Figure 25 As in Figure 20, except for sea-level pressure 
Gillon a 


Table 9 is a list of the values of the six parameters 
for this case (Tables 3 and 4) compared to the overall mean 


values for the developer and false-alarm classifications. 
oe 


The values of four of the six parameters for this case were 
closer to the mean values for false alarms. As defined by 
Fgquation (6) in Chapter II, Section D, the distance between 
each value and the midpoint of the two classification 
groups for each parameter contributes to the computation of 
the probabilities of group membership. As calculated by 
the discriminant analysis, the probability that this case 
belongs to Group 1 is 0.0003 and the probability that it 
belongs to: Group -2 is 49997, LAS <Grscriaminanke Lunctron 


therefore correctly classified this case as a false alarm. 


ike Dale <9 Ex ‘peremeters. for case: UKM 20000707 iz. Ae Ol) 
(third row) compared to the mean values for 
developers (first row) and false alarms (second 
row). The fourth and fifth rows contain the 
differences between the values for this case 
and the mean values for developers and false 
alarms, respectively. The smaller differences 
are highlighted. 


pera eipes 

developers 

Mean for false 
eee 





Incorrectly Classified False Alarm 
This section examines a tropical vortex that was 
initially identified in the UKMET 48-h forecast at 18°N, 
2PM. cat; WOUO0WEC. Lo. September 200s 4™URM 2O050 919: Oe 002" px 
The 48-h UKMET forecast to be examined was initiated at 


1200 UTC 23 September 2005. Vorticity and warm core values 


op) 


for this vortex are identified on Figure 17/ with a green 
arrow. The sequence of forecast tracks for this vortex is 


Shown in Figure 26. The 48-h forecast track is illustrated 


in Figure 2/7. 





Figure 26 The analyzed track (black) and all UKMET 
forecast tracks (colors) re the case 
UKM. 20050019 Ole 002. 





Figure 27/7 The analyzed track (black) and UKMET forecast 
track initiated at 1200 UTC 23 September 2005 
holue) fot Case UKM -2Zo0500L9 UL 002; Tie 45 
h analyzed and forecast positions are circled 
in red. 
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The UKMET forecast and analyzed values for the six 
parameters used in the 48-h discriminant function (Tables 3 
and 4) are compared in Figures 28-33. Figure 28 
illustrates that the 48-h forecast relative vorticity was 
higher than the analyzed relative vorticity. Figure 29 
illustrates that the 48-h forecast warm core was lower than 
the analyzed warm core. Figure 30 illustrates that the 
48-h forecast wind shear was lower than the analyzed wind 
shear. Figure 31 illustrates that the 48-h forecast 
vertical motion was higher than the analyzed vertical 
motion. Figure 32 illustrates that the 48-h forecast vapor 
pressure was higher than the analyzed vapor pressure. 
Figure 33 illustrates that the 48-h forecast sea-level 


pressure was equal to the analyzed sea-level pressure. 





UKM 2005091900 018 002: 850-500 hPa Relative Vorticity 
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Figure Zo As in Figure VAC except for case 
UBM. 2O00D0219 018 02. The UKMET 4A8-h 
forecast values from 1200 UTC 23 September 
2005 are plotted with red circles while 
analyzed values are plotted with blue squares. 
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UKM 2005091900_018_002: 700-500 hPa Warm Core 
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Figure 29 As in Figure 28, except for warm core (°C) at 
700-500 hPa. 
UKM 2005091900 018 002: 200-850 hPa Vertical Wind Shear 
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Figure 30 As in Figure 28, except for vertical wind 


shear (m s+) between 200 and 850 hPa. 


oe 











UKM 2005091900_018 002: 500 hPa Vertical Motion 
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Figure 31 As in Figure 28, except for vertical motion 
(m s~ * 10) at 500 hPa. 








UKM 2005091912 018 002: 700-500 hPa Vapor Pressure 
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Figure 32 As in Figure 28, except for vapor pressure (mb) 
between 700 and 500 hPa. 
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UKM 2005091912 018 002: Sea-level Pressure 
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Figure 33 AS in Figure 28, except for sea-level pressure 
(mb). 


Table 10 is a list of the values of the six parameters 
for this case compared to the overall mean values for the 
developer and false-alarm classifications. Values of five 
of the six parameters for this case were closer to the mean 
values for developers than to the mean values for false 
alarms. As calculated by the discriminant analysis, the 
probability that this case belongs to Group 1 is 0.84 and 
the probability that it belongs to Group 2 is Q.16. The 
discriminant function therefore incorrectly classified this 
case as a developer. The cause of this misclassification 
can be seen in Figures 28, 30, and 31 which demonstrate 
that through the 48-h forecast there waS a strong 
indication that the vortex would develop into a tropical 
cyclone. While the model would eventually weaken the 
vortex beyond 48h, it did so 12h after the actual vortex 


weakened. 
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Tabike 10 SLX Parameters- For. Case UKM. 20050919 016002 
(third row) compared to the mean values for 
developers (first row) and false alarms (second 
row). The fourth and fifth rows contain the 
differences between the values for this case 
and the mean values for developers and false 
alarms, respectively. The smaller differences 
is highlighted. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 


As the NHC and JTWC issue tropical cyclone advisories 
for longer forecast periods, the necessity to predict 
tropical cyclone formation becomes more important. Many of 
the processes that are believed to control the timing of 
tErOpreal cyclone formation Sccur aime the mesoscale. 
Therefore, it 1S a matter of current debate as to whether 
global model forecasts contain valuable information with 
regard to tropical cyclone formation. However, the current 
circumstances are such that the global models are the only 
numerical data routinely available to forecasters for this 
type of prediction. Therefore, the goal of this study was 
to use the global model data available in the TCVTP 
database to develop a tool to aid in the prediction of 


tropical cyclone formation. 


After a preliminary examination of forecasts from the 
three models, it became clear that the models could not 
effectively distinguish developing vortices from the large 
number of non-developing vortices (Appendix B). As a 
result, the focus was shifted to only those cases in which 
a model forecasted a vortex to exceed vorticity and warm- 
core thresholds for tropical cyclone formation. Hence, the 
objective was to develop a procedure for determining which 
model-predicted tropical cyclones will actually develop and 


which are false alarms. 


In this study, nine parameters relevant to tropical 
cyclone formation were subjected to a linear discriminant 
analysis. The nine parameters derived from global model 
OUTDUL. “do Mot. define: all Eropical -cyclone. characteristics. 


VOrEICeS: are adentarred. aml. which verivcrey and ‘warm. core 
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thresholds are exceeded, but in which other parameters 
(e.g., wind speed) suggest that the vortex is not a 
Eropucal cyclone. Correct forecasts of vorticity and warm 
core parameters for these types of vortices are not 
considered false alarms since the forecasts did verify in 


the correct category. 


The discriminant analysis was performed on the nine 
parameters from each model at each forecast period (30 
analyses Mr “Portal to determine the combination of 
parameters that best discriminates between a vortex that 
was correctly forecast to intensify into a tropical cyclone 
(developer) and a vortex that was forecast to intensify 
EMCO: «a “Topical eyeclone,. bul. Guid. Tol. (false <alarm)-. The 
performance of the resulting discriminant functions were 
then assessed using the Heidke Skill Score and Receiver 
Operating Characteristic curves. 

A. CONCLUSIONS 

The prameary conclusion of This ‘thesis: 2s thar lanear 
discriminant analysis applied to the parameters defined in 
the Tey P database provides a Wa. acd. method oun 
discriminating between developers and false alarms in the 
global model data. The resulting discriminant functions 
exhibited various levels of skill. In 27 of 30 cases, the 
discriminant function performed better than a random 
forecast as measured by the HSS and the area under the ROC 
curve. It 1S important to note that the VORTRACK system 
and the TCVTP database make this type of data analysis 


quick and simple to complete. 


Efforts to reduce the number of parameters considered 
in the analysis (by considering PDFs, for example), did not 


yield profitable results. The discriminant functions 
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resulting from each of the thirty analyses used different 
combinations of the nine parameters (Table 3). Lt 2s 
therefore concluded that all possible combinations of all 
available parameters should be considered in a discriminant 


analysis. 


Some challenges remain to performing this type of data 
analysis. Models undergo frequent updates. Fach time 
changes are made to the model, a new, and sufficient, 
training sample would be required before the discriminant 
analysis could yield meaningful results. Additionally, the 
small number of developing vortices identified and tracked 
by the TCVTP at the longer forecast intervals poses a 
problem. For example, not enough developing vortices were 
tracked in the 2005 UKMET forecasts for the discriminant 


analysis to produce meaningful results beyond 84h. 


The discriminant function applied to the UKMET model 
outperformed those applied to NOGAPS and GFS in eight of 
the 10 forecast periods. This superiority is at least 
partially attributable to a low false-alarm rate associated 
with the UKMET model. However, the UKMET model also has a 
tendency for a low probability of detection which is 
responsible for the lack of cases at the longer forecast 
intervals. The GFS is the only model in which the TCVTP 
identifies sufficient vortices at all forecast intervals. 
However, the HSS and ROC curves for GFS suffer as a result 
of its tendency to over-predict formation (1.e., higher 
false-alarm rate). 

B. RECOMMENDATIONS FOR FUTURE WORK 

This analysis should be replicated for different 

datasets to include additional geographic regions and time 


periods. If future analysis demonstrates that the models 
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are becoming capable of discriminating between developers 
and false alarms, the methodology should be redirected 


toward that important forecast problem. 


Two extensions to the discriminant analysis may lead 
to improved results. First, all parameters in this study 
were considered in a static sense. However, the time 
variability, or trend, of parameters could be incorporated 
and may have great potential to increase the skill of the 
JESCriIminant, <aradysas. Second, the analysis could consider 
combinations of parameters from multiple models that would 
result in an ‘ensemble’ OWSCriIMinane,. Lume uo: Fox 
example, the discriminant function for the 12-h forecast 
period may include the vorticity parameter from the GFS 
model, the warm-core parameter from the NOGAPS model, and 
the vapor pressure parameter from the UKMET model. 
However, the probability of detection for the UKMET and GFS 
models would need to increase for an ensemble approach to 


be useful at longer forecast intervals. 
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APPENDIX A. SCATTERPLOTS FOR GFS 12-H FORECAST 
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(SF 12-h Forecast Avg. 50-500 hPa Yorticity and 7U0-500 hPa Vvarm Core 
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700-500 hPa Warm Core (°C) 


APPENDIX B. SCATTERPLOTS INCLUDING OMEGA 
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700-500 hPa Warm Core (°C) 
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(5F Yb-h Forecast Avg. 50-500 hPa Yorticity and 7U0-5U00 hPa Vvarm Core 


Now Dene bop 

Deve bop 

Develop aid + ongea 
Fancy 

Fad) 2 omega 
Fado 

FAdon &< omega 
Feb) 

FApotiys = omega 





HoU-5U0 hPa Relative Vorticity qo sy 


(SF 108-h Forecast Avg. ¢50-500 hPa Vorticity and #U0-5U0 hPa Vvarm Core 


Now Dene bop 

Deve hop 

Deve bp aid = ongea 
Fancy 

FAM) 2 omega 
Fado 

FAdon & = omega 
Feb) 

FA pots = omega 


H5U-5U0 hPa Relative Vorticity qo 4 
13 


(SF 120-h Forecast Avg. g50-500 hPa Vorticity and #U0-5U0 hPa Vvarm Core 
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NisP 24-h Forecast Avg. 650-500 hPa Vorticity and #UU-500 hPa VVarm Core 
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NisP 46-h Forecast Avg. 650-500 hPa Vorticity and #UU-500 hPa VVarm Core 
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NisP f2-h Forecast Avg. 650-500 hPa Vorticity and #UU-500 hPa VVarm Core 
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NisP 84-h Forecast Avg. 650-500 hPa Vorticity and #UU-500 hPa VVarm Core 
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NisP Y6-h Forecast Avg. 650-500 hPa Vorticity and #UU-500 hPa VVarm Core 
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NisP 106-h Forecast Avg. 850-500 hPa Yorticity and 7U0-500 hPa Vvarm Core 
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NisP 120-h Forecast Avg. 50-500 hPa Yorticity and 7U0-500 hPa Yvarm Core 
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UKM 24-h Forecast Avg. 650-500 hPa Vorticity and #UU-500 hPa VVarm Core 
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UKM: 24-h Palammese! Distributions 
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UKM: — h Pr alernerel Distributions 
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